For different clustering effects of Clustering by Fast Search and Find of Density Peaks (CFSFDP) on different datasets, an improved CFSFDP algorithm based on spectral clustering was proposed, namely CFSFDP-SA (CFSFDP based on Spectrum Analysis). Firstly, a high-dimensional non-linear dataset was mapped into a low-dimensional subspace to realize dimension reduction, then the clustering problem was transformed into the optimal partitioning problem of the graph to enhance the algorithm adaptability to the global structure of the data. Secondly, the CFSFDP algorithm was used to cluster the processed dataset. Combining the advantages of these two clustering algorithms, the clustering performance was further improved. The clustering results of two artificial linear datasets, three artificial nonlinear datasets and four real datasets in UCI show that compared with CFSFDP, the CFSFDP-SA algorithm has higher clustering precision, achieving up to 14% improvement in accuracy for high-dimensional dataset, which means CFSFDP-SA is more adaptable to the original datasets.